An Experimental Approach to Measuring Ideological Positions in Political Text∗

Though powerful as general tools, automated measures of position-taking in text often perform poorly when models of speech are difficult to develop or theoretically contested. Rather than model text, I develop an experimental approach to measure perceptions of partisanship in speech, with an application to 2008 Congressional advertisements. I randomly assign ads to subjects recruited in a large-N survey, and ask them to ‘guess’ the party of featured candidates, with ads scored as their average party inference. These party perception scores are empirically synonymous with a liberal-conservative dimension, and highly reliable across samples and experimental conditions. Party identity has little impact on guesses, indicating the inferential task significantly mutes partisan bias. For validation, I assess which words influence guessing, and whether ad-scores correspond to expectations about how candidates target voters. Importantly, this experimental approach can augment or validate automated text analysis, and generalize to study speech across many other contexts.